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Diagnosing  Knowtedge  States  in  Algebra 
Using  the  Rule  Space  Model 

Abstract 

This  paper  illustrates  the  use  of  rule  space  as  a  tool  to  support  cognitive  analyses  of  students' 
mathematical  behavior.  The  rule  space  approach  is  explained  and  is  then  used  to  classify  students 
into  erne  of  two  methods  for  solving  linear  algebraic  equations  in  one  unknown  and  to  diagnose 
their  knowledge  states  in  this  topic.  A  32-item  test  with  open-ended  questions  was  administered  to 
231  eighth  and  ninth  graders.  The  following  outctxnes  of  the  rule  space  model  were  presented:  (a) 
a  classification  of  examinees  into  knowledge  states  resulting  fiom  the  two  solution  approaches  at 
the  group  level  along  with  individual  examples;  (b)  tree-diagrams  of  the  transitional  relationships 
among  the  states  for  each  strategy.  Implications  for  using  the  feedback  provided  by  the  rule  space 
model  in  the  context  of  instruction  and  assessment  are  discussed. 
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Diagnosing  Knowledge  States  in  Algebra 
Using  the  Rule  Space  Model 

Indices  that  may  be  quickly  and  inexpensively  generated  fiiom  standard  mathematics  tests 
include  total  number  conect  for  each  student,  measures  of  central  tendency,  dispersion,  and 
standard  errors  of  measurement  While  these  summary  and  descriptive  statistics  are  of  value  in 
ranking  students  or  comparing  a  student's  performance  to  the  performance  of  students  on  some 
larger  normative  sample,  they  do  not  provide  much  diagnostic  information  about  the  mathematical 
operations  that  the  student  has  mastered  or  has  not  yet  mastered. 

In  the  case  of  solution  of  linear  equations  in  one  unknown,  for  example,  a  teacher  may 
wish  to  know  more  than  that  a  given  student  is  "poor  at  algebra"  because  his  or  her  score  was  one 
standard  deviation  below  the  class  mean.  Ideally,  die  teacher  would  wish  to  know  which  of  the 
many  components  of  performance  in  algebra  is  causing  difficulty  for  a  given  shident  and  fra*  the 
class  as  a  whole.  Adequate  performance  in  the  algebra  of  linear  equations  requires  more  than  skill 
in  applying  an  algorithm.  It  tests  upon  adequate  performance  in  and  understanding  of  a  larger 
body  of  mathematics  that  ranges  from  mastery  of  simple  operations  such  as  addition  to  mastery  of 
more  difficult  concepts  such  as  the  distributive  law  and  quotients.  Armed  with  this  diagnostic 
information,  the  teacher  may  then  examine  the  difficult  area(s)  for  the  students  in  terms  of 
misconceptions  or  faulty  skill  performance  using  any  of  the  interview  and  protocol  analytic  tools 
provided  by  researchers  in  cognitive  science.  In  considering  group-level  perform  jice,  the  teacher 
may  wish  to  examine  teaching  methods  to  determine  if  these  are  responsible  in  any  way  for  the 
students'  misleaming. 

The  value  of  a  diagnostic  profile  that  points  out  deficiencies  and  strengths  in  the  students' 
performance  in  mathematics  has  long  been  recognized  (VanLehn,  1982).  However,  the 
computational  problems  involved  in  "teasing  out"  the  dimensions  underlying  students' 
performance  are  formidable.  VanLehn  (1982)  noted  that  thousands  of  hours  of  work  by  trained 
experimenters  were  required  to  determine  students'  "bugs"  in  subtraction.  The  problem  is 
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exacerbated  when  the  instructor  has  available  only  the  student's  conectAncoiiect  score  on  each 
item,  or  has  little  time  to  deal  with  detailed  levels  of  assessment 

To  illustrate  the  combinatoric  problem  involved  in  producing  profiles  of  mastery/non 
mastery  on  task  subcomponents,  imagine  that  one  can  describe  the  solution  of  an  item  in  terms  of 
the  mastery  of  four  underlying  dimensions.  Thus,  it  may  be  argued  that  a  student  who  fails  this 
item  (who  is  not  guessing)  may  have  failed  to  master  all  four  dimensions,  ot  failed  to  master  any 
three,  or  any  two,  or  any  one.  For  a  problem  with  four  dimensions,  there  are  2^  -1  patterns  that 
could  account  for  an  incorrect  answer.  In  general,  for  an  item  with  k  dimensions,  there  ate  2^  -1 
patterns  that  could  account  for  an  inconect  answer.  As  the  number  of  dimensions  increases,  die 
number  of  patterns  to  consider  climbs  exponentially. 

Tatsuoka  develqied  the  rule  space  methodology  to  address  the  combinatoric  problem 
associated  with  diagnosis  of  mastery  of  underlying  dimensions  of  an  item  (called  attributes),  (e.g., 
Tatsuoka,  1983, 1985, 1990, 1991;  Tatsuoka  &  Tatsuoka,  1987).  An  attribute  of  a  task  is  a 
descripticHi  of  the  processes,  skills  or  knowledge  a  student  would  be  required  to  possess  in  order 
to  successfully  cmnplete  the  target  task.  Attributes  are  not  generated  by  rule  space;  they  are 
generated  by  a  domain  expert  (usually  in  concert  with  a  cognitive  scientist).  They  may  include, 
but  are  not  limited  to,  a  student's  ability  to  perform  some  procedures.  Attributes  may  also  include 
a  student's  use  of  heuristics,  or  adoption  of  a  strategy.  In  general,  rule  space  can  handle  any 
expressicm  of  an  underlying  dimension  of  a  task  that  can  be  specified  to  the  extent  that  certain  items 
tap  that  attribute  and  other  that  do  not.  By  examining  a  student's  differential  performances  on  the 
items,  rule  space  categorizes  students  into  the  attribute  mastery  pattern  that  would  best  account  for 
the  student's  individual,  item-response  pattern. 

For  the  mathematics  educator,  rule  space  can  provide  the  following  diagnostic  information: 
(a)  a  descripti(Hi  of  each  student's  mastery  (and  nonmastery)  of  the  attributes  judged  by  a  domain 
expert  to  be  necessary  for  successfully  completing  the  test;  (b)  a  description  of  group  level  mastery 
patterns  obtained  by  aggregating  across  the  individual  profiles;  and  (c)  partial-mastery  charts  that 
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can  be  used  to  aid  in  the  design  of  remediation  (Tatsuoka  &  Tatsuc^  1992).  A  detailed 
description  of  rule  space  is  beyond  the  scope  of  this  paper.  A  simplified  description  follows. 

Rule  space  is  a  statistical  methodology  for  classifying  students'  responses  to  a  set  of  items 
into  one  (or  more)  prespecified  attribute-masteiy  patterns.  In  practice,  a  domain  expert  and 
cognitive  scientist  would  identify  the  attributes  of  the  target  task  that  are  of  interest  They  would 
then  write  items  that  sample  from  this  set  of  attributes.  The  resulting  items  and  attributes  would 
then  be  arranged  in  an  attribute-by-item  matrix  (referred  to  as  a  O  matrix  in  rule  space). 
Unformnately,  a  student's  actual  mastery  or  nonmastery  of  a  set  of  attributes  cannot  be  measured 
directly,  but  must  be  inferred  from  the  student's  pattern  of  responses  to  the  items.  In  an  ideal  case, 
a  student  who  had  mastered  some,  but  not  other,  attributes  would  answer  correctly  only  those 
items  that  contain  attributes  that  he  or  she  had  mastered,  and  answer  incorrectly  those  items  that 
contain  at  least  one  attribute  that  he  or  she  had  not  mastered.  Such  a  student  would  produce  an 
ideal  item-response  pattern.  Within  rule  space,  specialized  functions,  called  Boolean  Description 
Functions  (BDF),  are  used  systematically  to  determine  the  knowledge  states  of  interest  (i.e.,  those 
that  describe  ideal  behavior  in  terms  of  attributes)  and  to  map  them  into  ideal  item-response  patterns 
(Tatsuoka,  1991;  Varadi  &  Tatsuoka,  1989).  Rule  space  then  plots  the  ideal  item-response 
patterns  in  terms  of  two  variables;  9  (theta),  and  ^  (zeta). 

6  and  The  ability  continuum  derived  from  an  item-response  (IRT)  analysis  (Lord  & 
Novick,  1968),  6,  is  used  as  one  dimension  along  which  to  describe  the  ideal  item-response 
patterns.  Thus,  a  high-ability  student  (scoring  high  on  6)  would  have  an  ideal  item-response 
pattern  with  many  Is  and  few  Os  (for  correct  and  incorrect  responses  to  items,  respectively); 
conversely,  a  student  at  the  lower  end  of  the  ability  continuum  (scoring  low  on  6)  would  display 
an  ideal  item-response  pattern  containing  mostly  Os.  A  student  of  high  ability  who  gets  some  easy 
items  inccvrect,  or  a  student  of  low  ability  who  gets  some  hard  items  correct  would  be  measured 
high  on  an  "unusualness  of  response"  scale,  which  is  what  ^  is  (Tatsuoka,  1984;  Tatsuoka  & 

Linn,  1983).  C  is  the  second  dimension  that  rule  space  uses  to  describe  students'  responses. 
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Thus,  rule  space  generates  a  two-dimensional  coordinate  space  (with  6  on  the  x-axis  and  ^ 
on  the  y-axis)  in  whose  plane  certain  points  represent  the  6  and  C  of  the  ideal-response  patterns. 
However,  students'  performances  on  the  test  items  are  often  subject  to  fluctuations.  Producing  an 
ideal  response  pattern  is  likely  to  be  rare.  Students'  item-response  patterns  that  deviate  from  an 
ideal  response  pattern  are  considered  as  "fuzzy"  response  patterns.  Points  corresponding  to  the 
fuzzy  response  patterns  swarm  around  their  respective  ideal  response  patterns,  and  generate 
regions  within  probability  ellipses  with  the  ideal  response  patterns  as  their  centers.  A  90% 
probability  ellipse  encloses  90%  of  the  fiizzy-response-pattem  points;  a  95%  probability  ellipse 
encloses  95%  of  them;  and  so  forth.  Rule  spatte  then  uses  information  on  a  student's  actual  score, 
measured  on  G,  and  C,  to  decide  where  in  the  two-dimensional  space  spanned  by  these  measures 
the  student's  fuzzy  item-response  pattern  lies  (Tatsuoka  &  Tatsuoka,  1987).  A  student  is 
classified  to  the  ideal  response  pattern  that  embraces  his  or  her  point  in  the  smallest  associated 
ellipse.  This  determination  is  made  by  measuring  how  far  from  the  centroid  the  student's  point  is, 
in  terms  of  Mahalanobis'  distance.  Once  the  most  likely  ideal  item-response  pattern  is  identified, 
the  most  conservative  attribute-mastery  pattern  for  that  ideal  item-response  pattern  is  assigned  by 
rule  space  to  that  student  The  most  conservative  pattern  is  chosen  for  instructional  purposes.  The 
most  conservative  pattern  will  err  in  the  directicHi  of  suggesting  that  a  student  has  not  mastered  the 
identified  attributes,  when  he  or  she  may  have  mastered  them.  Thus,  the  conservative  diagnosis 
would  spur  a  remedial  strategy  that  would  be  most  likely  to  target  the  student's  weaknesses. 

Rule  space  entails  a  statistical  pattern  classification  approach.  Its  accuracy  of  classification 
depends  on  how  well  the  items  are  written,  how  well  they  test  (as  unambiguously  as  possible)  the 
attributes  that  were  established  by  the  domain  expert,  and  the  amount  of  error  in  the  student's 
responses.  Since  rule  space  does  not  produce  the  attributes,  the  onus  lies  on  domain  experts  and 
cognitive  scientists  to  provide  it  with  useful  descriptions.  For  areas  that  are  well-defmed  (e.g., 
subtraction  of  fractions,  signed  numbers  operations),  rule  space  has  been  shown  to  perform  quite 
well  (Tatsuoka,  1990;  Tatsuoka  &  Tatsuoka,  1992). 
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From  among  the  several  methods  possible  to  solve  a  given  linear  equation  in  one  unknown, 
we  have  chosen  to  demonstrate  the  use  of  rule  space  using  two  different  ^proaches  (expressed  as 
two  different  Q  matrices).  One  method  involves  the  use  of  a  simple  heuristic  -  initially  evaluating 
the  equation  to  determine  if  a  simpler  solution  path  would  result  by  not  rewriting  the  equation  in 
standard  form  until  the  rinal  step  (method  I)*  'The  other  method  involves  consistently  rewriting  the 
equation  in  standard  form  (i.e.,  with  variables  on  the  left-hand  side  of  the  equation  and  constants 
on  the  right  (method  II).  More  details  regarding  the  two  methods  are  given  in  the  method  sectitm. 
The  purpose  of  the  present  study  was  to  illustrate  the  application  of  the  rule  space  model  for 
diagnosing  students'  knowledge  states  in  linear  equations  based  on  the  two  specified  solution 
methods.  Thus,  we  will  see  how  rule  space  can  be  used  to  identify  students  who  may  need  further 
remediation,  to  identify  subcomponents  of  linear  algebra  that  may  be  causing  difficulty  for  the 
entire  group  of  students,  to  produce  partial  mastery  charts  that  may  form  the  basis  of  fruitful 
remediation,  and  to  identify  students  for  whom  it  may  be  of  value  to  study  further  in  terms  of  their 
particular  strategy  use. 

Method 

Subjects 

The  sample  consisted  of  231  8th  and  9th  graders  (age  14-15)  from  an  integrative  high 
school  in  Tel  Aviv.  Fifty-seven  percent  of  the  subjects  were  girls.  The  students  studied 
mathematics  in  high  and  low  achievement  groupings  (106  in  the  former  and  125  in  the  latter). 
Instruments  and  procedures 

A  32-item  diagnostic  test  in  linear  algebraic  equations  in  one  unknown  was  developed  by 
Gutvirtz  (1989).  (The  test  items  appear  in  Appendix  A). 

The  internal  consistency  of  the  32-item  test  as  measured  by  Cronbach's  Alpha  coefficient 
was  0.95.  The  item  difficulty  indices  (percent  correct)  ranged  from  0.41  to  0.93  with  an  average 
of  0.74.  The  item  discrimination  indices  (item-total  correlations)  ranged  from  0.40  to  0.75  with  an 
average  of  0.60. 
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Two  sets  of  attributes  were  specified  fat  the  two  solution  methods  (see  Tables  1  and  2)  and 
these  sets  used  to  produce  two  separate  Q  matrices  (see  Appendices  A  &  B).  The  two  sets  of 
attributes  result  fiom  a  strategic  decision  made  at  the  outset  In  method  I,  a  heuristic/'evaluation," 
is  applied,  wherein  the  student  scans  the  equation  in  its  initial  form  to  determine  if  it  is  likely  to  be 
simpler  to  delay  writing  the  equation  in  standard  form  until  the  fiiuil  step.  For  example,  the 
evaluation  rule  could  be  implied  to  item  5  in  the  test  When  the  evaluation  heuristic  is  tqrplied  the 
solution  unfolds,  thus: 

4x  +  21=  lOx  +  17 

21  - 17  =  lOx  -  4x  (evaluating,  and  subtracting  an  x-term  and  a  constant  from  both  sides) 
4  =  6x  (adding  or  subtracting  variable  terms) 

6x  =  4  (applying  the  symmetry  law) 

X  =  4/6  (dividing  across  by  the  coefficient  of  x,  when  a>b) 

These  operations  are  denoted  in  Table  1  as  12, 2, 6, 1 1, 10,  and  14  (see  also  the 
corresponding  row  for  item  S  in  Appendix  A). 

In  method  11,  the  student  performs  the  mathematical  operations  necessary  to  bring  the  x- 
terms  to  the  left-hand  side  of  the  equation,  and  the  constants  to  the  right  in  all  cases.  Thus  when 
the  evaluation  heuristic  is  not  applied,  the  solution  path  is  more  complex,  since  it  now  involves 
operations  with  signed  numbers: 

4x  +  21  =  lOx  +  17 

4x  - 10  X  =  17  -  21  (subtracting  an  x-term  and  a  constant  from  both  sides) 

-6  X  =  -4  (adding  or  subtracting  variable  terms  and  operations  with  signed 

numbers) 

6x  =  4  (multiplying  both  sides  of  the  equation  by  -1) 

X  =  4/6.  (dividing  across  by  the  coefficient  of  x) 

These  operations  are  denoted  in  Table  2  as  2, 7,  8, 1 1, 13  (see  also  the  corresponding  row 


for  item  5  in  Appendix  B). 
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The  rule-space  analysis: 

1.  The  adequacy  of  the  two  attribute  matrices  was  tested  by  regressing  the  vector  of  item 
difficulties  on  the  set  of  attribute  vectors.  The  entire  set  of  attributes  accounted  for  95%  of  the 
variance  (R2=.95;  R^adj=-91)  for  method  I,  and  77%  of  the  variance  (R2=.77;  R2gjj=.63)  for 
method  n  in  the  toml  sample. 

2.  The  BELOG  program  (Mislevy  &  Bock,  1983)  was  used  for  estimating  the  item 
parameters  (g's  and  b's)  of  the  IRT  two-parameter  logistic  model.  The  g  values  ranged  from  O.SS 
to  2.20  with  a  mean  value  of  1.21;  the  b  values  ranged  from  -2. 12  to  0.45  with  a  mean  value  of 
-0.84. 

3.  In  order  to  determine  the  ideal  item-response  patterns  corresponding  to  the  attribute 
mastery  patterns,  the  BUGLIB  program  (Varadi  &  Tatsuoka,  1989)  was  used.  As  a  result,  461 
ideal  item-response  groups  (representing  461  different  knowledge  states)  were  generated  for 
method  I,  and  453  for  method  n. 

Results 

A.  Method  I  classification  results 

The  classification  of  the  actual  students'  response  patterns  into  the  461  predetermined 
knowledge  states  resulted  in  55  non-empty  groups.  A  summary  of  the  classification  results  is 
presented  in  Table  3.  As  can  be  seen  in  the  table,  15  groups  had  frequencies  of  2  or  more,  the 
maximum  having  10  students  in  a  group.  The  table  also  presents  the  states  into  which  one  or  more 
students  were  classified,  ordered  by  IRT  6.  Figure  1  is  a  tree  representation  of  those  states.  Each 
state  is  represented  by  a  node  indicating  the  non-mastered  attributes  in  that  knowledge  state,  and 
located  on  the  IRT  6-value  scale,  which  is  given  on  the  left  side  of  the  table.  The  arcs  connecting 
the  nodes  indicate  transitional  relationships  among  the  states.  A  transition  from  one  state  of 
knowledge  to  another  is  said  to  be  possible  whenever  the  set  of  non-mastered  attributes  associated 
with  the  second  state  is  a  proper  subset  of  the  first  state.  Thus,  arcs  connect  lower  knowledge 
states  to  higher  ones,  where  a  higher  state  is  defined  as  a  state  having  at  least  one  less  non- 
mastered  attribute  than  the  lower  state  connected  to  it. 
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Insert  Table  3  and  Figure  1  about  here 


B.  Method  B  classification  results 

The  classification  of  the  students'  response  patterns  into  the  453  predetermined  knowledge 
states  resulted  in  5 1  non-empty  groups.  A  summary  of  the  classification  results  is  presented  in 
Table  4.  As  can  be  seen  in  the  table,  20  groups  had  frequencies  of  2  or  more,  the  maximum  having 
8  students  in  a  group.  The  groups  are  ordered  by  IRT  6.  Figure  2  L  a  tree  representation  of  those 
states. 


Insert  Table  4  and  Figure  2  about  here 


C.  Classifying  Students  into  the  two  Solution  Methods. 

A  decision  rule  was  set  to  determine  which  of  the  two  methods  a  given  student  was  more 
likely  to  have  used.  The  shorter  of  the  two  distances  (Mahalanobis’  distances)  between  a  student's 
response  pattern  and  that  of  the  nearest  ideal  item-response  group  in  each  method  was  chosen  to 
indicate  the  student's  group  affiliation.  Applying  this  decision  rule  resulted  in  104  students  being 
classified  into  method  I  and  89  into  metliod  n.  Of  the  rest,  13  students  had  identical  Mahalanobis' 
distances  for  both  methods;  19  answered  all  items  correctly,  and  6  answered  all  items  incorrecdy  - 
thus  the  method  used  by  these  students  could  not  be  determined.  The  students'  average 
ability^roficiency  levels  as  measured  by  IRT  0  were  -0.08  and  0.05  for  methods  I  and  H, 
respectively  (with  SDs  0.98  and  0.84,  respectively).  Thus,  the  difference  between  the  two  groups 
in  mathematics  ability  as  inferred  form  their  performance  on  the  current  test  was  insignificant. 
Among  the  students  who  were  classified  into  methods  I  and  n,  80%  and  81%,  respectively,  were 
within  the  95%  probability  ellipses  of  knowledge  states-the  ideal  response  patterns  (x^cv  3df  aos 
=  0.35).  (For  a  complete  discussion  of  probability  ellipses  in  this  regard  see  Tatsuoka  & 
Tatsuoka,  1987). 
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Examples  of  Classified  Responses  for  Method  L 

To  illustrate  the  outcomes  of  the  rule  space  model  for  method  I,  three  students  who  were 
better  classified  to  this  method  are  now  described. 

Student  13  correctly  answered  6  items  (items  6, 9, 14, 17, 23, 29)  and  erred  on  26  items.  This 
student  was  classiHed  into  knowledge  state  No.  437  with  a  Mahalanobis'  distance  of  0.(X), 
indicating  a  perfect  match  between  the  studenf  s  response  pattern  and  the  ideal  response  pattern 
represented  by  that  knowledge  state.  As  can  be  seen  in  Table  3  the  IRT  6  value  for  that  state  is  a 
low  -1.73,  and  it  is  characterized  by  non-mastery  of  the  following  attributes  (see  Table  1);  1 
(adding  a  term  to  both  sides  of  the  equation),  2  (subtracting  a  term  from  both  sides  of  the 
equation),  3  (applying  order  of  operations),  4  (applying  the  distributive  law),  5  (applying  the 
commutative  law),  7  (applying  signed  numbers  operations),  8  (dividing  across  by  the  coefficient  of 
X,  resulting  in  x  =  b/a,  when  a  =b),  10  (dividing  across  by  the  coefficient  of  x,  resulting  in  x  =  b/a, 
when  a  >b),  12  (evaluating  the  equation),  13  (applying  order  of  operations  and  the  distributive 
law),  and  14  (applying  the  symmetry  law  and  evaluating  the  equation). 

In  order  to  reach  state  No.  0  (mastery  of  all  attributes)  from  the  state  the  student  is  currently 
in  (state  437),  a  number  of  transitions  need  to  take  place,  as  can  be  seen  in  the  tree  diagram 
presented  in  Figure  1  one  possible  path  is  through  states  429  (in  which  students  have  mastered 
attributes  1, 2, 4, 12,  and  14),  to  state  244  (attributes  3  and  5),  to  state  3  (attributes  8  and  10),  to 
state  1,  by  which  time  one  attribute  remains  to  be  mastered  (7),  thus  reaching  a  mastery  of  all 
required  skills  (state  0). 

Student  SO  correctly  answered  31  items  and  erred  on  item  21.  This  student  was  classified 
into  knowledge  state  No.  1  with  a  Mahalanobis'  distance  of  0.(X).  As  can  be  seen  in  Table  3,  five 
other  students  were  classified  into  this  knowledge  state,  which  has  a  6  value  of  1.23  and  is 
characterized  by  non-mastery  of  only  attribute  7  [Performing  signed  number  operations]. 

Student  175  correctly  answered  25  items  and  erred  on  items:  1, 13, 16, 19, 21, 25, 30. 
This  student  was  classified  into  knowledge  state  No.  301  with  a  Mahalanobis'  distance  of  0.04 
(the  student's  point  is  within  the  99%  probability  ellipse  for  that  state).  As  can  be  seen  in  Table  3, 
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six  other  students  were  classified  into  this  state  which  is  characterized  by  non-mastery  of  the 
attributes  1  [adding  a  term],  3  [order  of  operations],  7  [signed  number  operations],  and  13  [order 
of  operations  and  distributive  law].  In  order  to  reach  state  No.  0  (mastery  of  all  attributes)  firom  the 
state  the  student  is  currently  in,  a  number  of  transitions  need  to  take  place,  as  can  be  seen  in  the  tree 
diagram  presented  in  Figure  1.  One  possible  route  is  through  states  3, 1  to  state  0. 

Examples  of  aassified  Responses  for  Method  n 

To  illustrate  the  outcomes  of  the  rule  space  model  for  method  n,  three  students  who  were 
better  classified  to  this  method  using  the  above  decision  rule  are  now  described. 

Student  148  correcdy  answered  27  items  and  erred  on  5  items  (items  1, 13, 16, 25,  and  30). 
This  student  was  classified  into  knowledge  state  No.  234  with  a  Mahalanobis'  distance  of  0.00, 
indicating  a  perfect  match  between  the  student's  response  pattern  and  the  ideal  response  pattern 
represented  by  that  knowledge  state.  As  can  be  seen  in  Table  4  the  IRT  6  value  for  that  state  is  .51 
and  it  is  characterized  by  non-mastery  of  attribute  3  (see  Table  2):  (applying  order  of  operations). 
As  can  be  seen  in  Appendix  B,  61%  of  the  subjects  in  method  n  group  mastered  that  attribute. 

Student  136  correcdy  answered  26  items  and  erred  on  the  following  6  items  [items  3, 5, 

26, 27, 28,  and  31].  This  student  was  classified  into  knowledge  state  No.  59  with  a  Mahalanobis' 
distance  of  0.05  (i.e.,  the  student's  point  is  within  the  99%  probability  ellipse  for  that  state).  As 
can  be  seen  in  Table  4,  four  other  students  were  classified  into  this  knowledge  state,  which  has  an 
IRT  0  value  of  -0.01  and  is  characterized  by  non-mastery  of  attribute  1 1  (dividing  across  by  the 
coefficient  of  x,  resulting  in  x  =  b/a,  when  a  >b).  As  can  be  seen  in  Appendix  B,  57%  of  the 
subjects  in  method  II  group  mastered  that  attribute. 

Student  142  correcdy  answered  1 1  items  (items:  2,  6,  8,  14,  17,  18,  20,  22,  23, 24,  29) 
and  erred  on  the  other  items.  This  student  was  classified  into  knowledge  state  No.  83  with  a 
Mahalanobis'  distance  of  0.09  (a  value  within  the  99%  probability  ellipse  for  that  state).  This  state 
is  characterized  by  non-mastery  of  the  attributes  1  (adding  a  term  to  both  sides  of  the  equation),  4 
(applying  the  distributive  law],  6  (applying  the  distributive  ahd  commutative  laws),  8  (applying 
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signed  niunber  operatitms),  and  1 1  (dividing  across  by  the  coefficient  of  x,  resulting  in  x  =  b/a, 
when  a  >b).  No  other  student  in  our  sample  was  classiHed  into  that  state. 

D.  Comparing  the  Results  of  the  two  Solution  Methods. 

The  two  methods,  I  and  n,  yielded  overall  significandy  different  results  for  item  difficulties 
as  was  indicated  by  a  discriminant  analysis.  Thirty  five  percent  of  the  variance  in  item  difficulty 
was  explained  by  group  affiliation  to  method  I  or  n  (Wilks  Lamda  0.65,  32df  =  76.01, 
p<.(XX)l).  The  discriminant  function  yielded  substantive  weights  (>3)  for  the  following  items:  8 
(.62),  16  (.57),  18  (.35),  21  (.55),  27  (-.33),  28  (-.34) ,  29  (-.30).  (The  values  in  the  parentheses 
are  the  standardized  canonical  discriminant  fimcticm  coefficients).  As  is  evident  from  the  signs  of 
these  weights,  some  items  turned  out  to  be  easier  for  method  I  students  and  others  for  method  n 
students.  Item  difficulties  (percent  correct)  fw  each  method  appear  in  Appendices  A  and  B. 

The  mastery  level  for  the  two  groups  also  differ  as  can  be  seen  by  comparing  the  mastery  level  of 
similar  attributes  in  the  two  groups  given  in  Appendices  A  and  B.  These  differences  can  not  be 
tested  statistically  because  even  for  the  same  attribute  definition  different  items  may  apply  in  the 
two  methods.  However,  a  qualitative  comparison  of  the  interpretations  based  on  mastery  profiles 
for  each  method  indicates  that  for  students  in  method  I  the  least  mastered  attributes  (see  table  1)  are 
7  (Performing  signed  numbers),  13  (Applying  both  arithmetic  order  and  the  distributive  law  in  the 
same  equation),  10  (Dividing  across  by  the  coefficient  of  x,  [x=b/a  when  a>b])  and  5  (Applying 
the  commutative  law);  whereas  fm*  method  n  students  (see  Table  2),  the  least  mastered  attributes 
are:  6  (Applying  the  distributive  and  commutative  law),  4  (Applying  the  distributive  law),  1 
(Adding  a  term  to  both  sides  of  the  equation),  and  1 1  (Dividing  across  by  the  coefficient  of  x, 
when  a>b). 

Discussion 

This  paper  illustrated  the  use  of  rule  space  to  diagnose  student's  individual  and  group-level 
mastery  of  attributes  related  to  linear  algebra.  Two  different  pre-specified  solution  models  were 
identified  and  students  were  classified  according  to  them.  One  model  was  chosen  to  be  more 
mathematically  sophisticated  and  involves  mastery  of  the  symmetry  law  and  the  application  of  a 
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heuristic  that  allows  for  strategic  decision  making  when  solving  the  equation  (i.e.,  to  delay  writing 
the  equation  in  standard  from  imtil  the  final  step).  The  other  model  represents  a  solution  that 
progresses  in  a  more  standard  fashion  in  which  all  the  x-terms  are  brought  to  the  left-hand  side  of 
the  equation,  and  the  constants  to  the  right  Many  other  solution  models  could  exist  of  course.  In 
order  to  test  these  models,  unique  Q  matrices  would  have  to  be  written. 

Of  this  sample  of  students,  104  were  more  likely  to  be  using  the  heuristic  approach,  and  89 
students  the  standard  approach.  Supporting  evidence  for  this  distinction  was  found  in  that  item 
difficulties  differed  for  each  Q  matrix,  indicating  that  the  difficulty  of  an  item  is  a  function  of  the 
strategy  used  to  solve  it  (since  different  attributes  are  called  upon  for  each  method).  For  example, 
attribute  7  (Performing  signed  numbers,  negative  subtraction  and  multiplication  operations)  posed 
the  greatest  difficulty  for  students  classified  as  using  the  heuristic  approach.  This  finding  seems 
reasonable  in  that  students  who  evaluate  the  equation  to  see  whether  it  is  easier  (i.e.,  results  in 
positive  integer  values)  to  bring  x-terms  to  the  right-hand  side  rather  than  to  the  left-hand  side  of 
the  equation  would  generally  not  encounter  qierations  involving  negative  numbers.  Note  that 
attribute  7  poses  difficulty  across  all  levels  of  ability  (see  Table  3).  Attribute  13  also  poses 
consistent  difficulty  (Applying  both  arithmetic  order  and  the  distributive  law  in  the  same  equation). 
On  the  other  hand,  attribute  2  (Subtracting  a  term  from  both  sides  of  the  equation)  causes  difficulty 
for  only  the  lowest  ability  students.  For  students  using  the  standard  approach,  on  the  other  hand, 
attribute  6  (Applying  the  distributive  an  commutative  laws  in  the  same  equation)  proved  the  most 
difficult 

When  we  consider  the  partial-mastery  chart  for  students  using  the  heuristic  method  (Figure 
1),  we  see  how  the  transitional  states  are  interrelated  when  they  are  linked  as  proper  subsets  one  of 
the  other.  One  approach  to  remediation  using  this  chart  is  to  first  identify  the  knowledge  state  that 
best  describes  the  target  student.  Then,  to  consider  the  transition  path  that  causes  the  least  change 
as  reflected  on  the  ability  measure,  6.  Thus,  a  student  classified  to  state  437  is  more  likely  to 
respond  to  remediation  that  results  in  attaining  state  429  (i.e.,  remediating  attributes  1, 2, 4, 12, 
and  14),  rather  than  to  remediation  that  results  in  attaining  state  303  (i.e.,  remediating  attributes  2, 
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5, 8, 10, 12,  and  14)  —  since  the  latter  state  is  associated  with  higher-ability  students.  For  a  more 
complete  description  of  how  to  use  transitional  states  for  remediation  purposes,  see  Tatsuoka  and 
Tatsuoka  (1992). 

At  the  whole-class  level,  a  teacher  using  the  current  analyses  would  know  that  a  significant 
number  of  students  were  most  likely  not  using  the  heuristic  method.  Therefore,  the  teacher  could 
explicitly  teach  the  evaluation  heuristic,  which  would  provide  the  students  a  choice  of  solution 
models,  and  would  make  algebra  seem  less  mechanical  and  more  mathematical.  Concerning  the 
class's  performance  on  each  attribute,  the  teacher  could  address  each  of  the  unmastered  attributes 
using  whole  class  instruction.  Similar  options  would  exist  at  the  individual  student  level,  in  which 
the  teacher  could  focus  on  the  strategy-level  decisions  that  the  student  is  making  or  on  remediation 
of  the  nonmastered  attributes. 

Comparing  the  rule  space  and  buggy  approaches.  In  recent  years,  cognitive  scientists  and 
psychometricians  have  contributed  to  the  effort  m  better  understand  mathematics  performance 
beyond  simple  indices  (e.g.,  Birenbaum  &  Tatsuoka,  1987;  Brown  &  Burton,  1978;  Matz,  1982; 
Sleeman,  1984;  Tatsuoka,  1990;  Tatsuoka  &  Tatsuoka,  1992;  VanLehn,  1990).  An  alternative 
modeling  approach  to  rule  space  is  the  buggy  approach,  in  which  diagnoses  are  generated  in 
response  to  the  student’s  errors  (Sleeman,  Kelly,  Martinak,  Ward  &  Moore,  1989;  Payne  & 
Squibb,  1990;  VanLehn,  1982).  Many  such  errors  may  be  "wild"  or  result  from  slips  (e.g., 
Sleeman  et  al.,  1989).  As  a  ccmsequence,  remediation  resulting  from  buggy  analysis  may  lead  the 
teacher  and  student  far  afield  from  the  target  task.  To  illustrate,  consider  an  equation  in  the  form  ax 
=  b.  Bugs  that  have  been  noted  for  this  case  generate  x  =  b  (Sleeman  et  al.,  1989),  x  =  b  -  a 
(Sleeman  et  al.,  1989;  Payne  &  Squibb,  1990),  x  =  -(a  +  b)  (Gutvirtz,  1989),  x  =  a  -  b  (Gutvirtz, 
1989),  and  x  =  a  +  b  (Gutvirtz,  1989;  Payne  &  Squibb,  1990).  To  explain  each  of  these  cases,  the 
teacher  must  make  complex  inferences  about  the  underlying  mathematical  models  of  the  student, 
and  design  remediation  targeted  to  these  inferences  ~  predicated  tm  the  questionable  assumption 
that  students  are  not  generating  many  of  these  errors  capriciously  (Sleeman  et  al.,  1989;  Payne  & 
Squibb,  1990). 
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The  rule  space  analysis,  by  contrast,  focuses  diagnosis  and  remediation  decisions  on 
attributes  that  are  integral  to  the  task  at  hand.  Then  rule  space  analysis  considers  the  extent  to 
which  the  attributes  for  a  given  item  are  mastered  over  the  entire  test  For  item  7  in  the  test  [8  + 

4(x  -  3)  =  24]  method  11,  for  example,  the  attributes  to  consider  for  this  item  would  be  1  (Adding  a 
term  to  both  sides  of  the  equation),  4  (Applying  the  distributive  law),  5  (Applying  the  commutative 
law),  6  (Applying  the  distributive  and  commutative  law),  8  (Performing  signed  numbers 
operations),  and  10  (Dividing  across  by  the  coefficient  of  x,  when  a<b  [x=b/a]).  The  decision  as 
to  which  attributes  would  be  remediated  would  be  based  not  on  the  given  student's  bug(s)  for  that 
item,  rather  on  an  analysis  of  how  the  attributes  were  mastered  across  the  entire  set  of  items  by  that 
student  In  addition,  the  information  gathered  on  the  entire  sample  of  students  allows  the  teacher  to 
consider  a  pathway  to  mastery  for  this  student  by  considering  the  number  of  students  assigned  to 
each  knowledge  state  (see  Table  4  and  Figure  2).  The  usefulness  of  remediation  based  on  these 
knowledge  states  remains  to  be  tested  empirically.  If  they  are  found  to  be  of  value  instructionally, 
remedial  strategies  can  be  proposed  and  scripted  beforehand  to  address  nonmastery  of  each  of  the 
attributes.  Further,  the  rule  space  analysis  permits  the  investigation  of  the  application  of  these 
attributes  at  a  strategic  level  (heuristic  vs.  standard  methods  in  this  case),  which  lends  itself  to 
remediation  at  this  level.  Finally,  a  careful  examination  of  the  Q  matrix  and  the  resulting  group 
attribute  mastery  profiles  can  aid  in  designing  future  tests  in  that  topic,  thus  increasing  the  validity 
of  those  tests.  Regarding  questions  of  validity,  it  should  be  noted  that  the  two  Q  matrices 
(describing  two  different  approaches  to  solving  the  linear  equations)  resulted  in  different  item 
difficulties. 
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Table  1.  Attributes  Used  to  Describe  Method  I. 

No.  Description 

1  Adding  a  term  to  both  sides  of  the  equation 

2  Subtracting  a  term  from  both  sides  of  the  equation 

3  Applying  arithmetic  oder  of  operations 

4  Applying  the  distributive  law 

5  Applying  the  oxnmutative  law 

6  Adding  or  subtracting  variable  terms 

7  Performing  signed  numbers,  negative  subtraction  and  multiplication  operations 

8  Dividing  across  by  the  coefficient  of  x,  [resulting  in  x=h/a  when  a=b] 

9  Dividing  across  by  the  coefficient  of  x,  [resulting  in  x=b/a  when  a<b] 

10  Dividing  across  by  the  coefficient  of  x,  [resulting  in  x=b/a  when  a>b] 

1 1  Applying  symmetry  law 

1 2  Evaluating  the  equation  to  determine  the  simplest  solution  path 

1 3  Applying  both  arithmetic  order  and  the  distributive  law  in  the  same  equation 

14  Applying  symmetry  law  and  evaluating  the  equation  to  determine  the  simplest  solution  path 
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Table  2.  Attributes  Used  to  Describe  Method  n. 

No.  Description 

1  Adding  a  term  to  both  sides  of  the  equation 

2  Subtracting  a  term  from  both  sides  of  the  equation 

3  Applying  arithmetic  order  of  operations 

4  Applying  the  distributive  law 

5  Applying  the  ctxnmutative  law 

6  Applying  the  distributive  and  commutative  law 

7  Adding  or  subtracting  variable  terms 

8  Performing  signed  numbers,  negative  subtraction  and  multiplication  operations 

9  Dividing  across  by  the  coefficient  of  x,  [resulting  in  x=b/a  when  a=b] 

10  Dividing  across  by  the  coefficient  of  x,  [resulting  in  x=b/a  when  a<b] 

1 1  Dividing  across  by  the  coefficient  of  x,  [resulting  in  x=b/a  when  a>b] 

12  Number  of  distinct  mathematical  operations  >  3 

13  Multiplying  both  sides  of  the  equation  by  (-1) 
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Table  3.  For  Method  I  the  States  with  two  or  More  Students  Qassified  into  Them  Ordered  by 
Theta  (6),  and  a  Listing  of  Attributes  Not  Mastered. 


State  No. 

iRTe 

Frequency 

Attributes  not  Mastered 

0* 

s.oo 

19 

(all  mastered) 

1 

1.23 

6 

7 

3 

.41 

6 

7.  13 

6 

.12 

3 

4.  7.  13 

11 

.02 

3 

4.  5.  7.  13 

86 

-.21 

3 

7.  10 

107 

-.32 

2 

1.  5.  7.  10 

180 

-.13 

2 

7.  8.  14 

244 

-.57 

10 

7.  8.  10.  13 

301 

.13 

7 

1.  3.  7.  13 

303 

-.08 

3 

1.  3.  4.  7,  13 

304 

.12 

2 

3.  5.  7.  13 

348 

-.59 

7 

3.  5,  7.  10.  13 

376 

-.24 

2 

3. 4.  5.  7,  8.  13 

429 

-.73 

5 

3.  5.  7.  8.  10.  13 

437 

-1.73 

3 

1.  2.  3.  4.  5.  7.  8.  10.  12,  13,  14 

372* 

-5.00 

6 

1.  2,  3.  4,  5.  6.  7.  8,  9.  10,  11.  12.  13.  14  (none  mastered) 

*  Students  in  these  states  were  not  included  in  the  analysis  since  their  method  could  not  be 
determined. 
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Table  4.  For  Method  H;  The  States  with  two  or  more  Students  Qassified  into  them  Ordered  by 
Theta  (6),  and  a  Listing  of  Attributes  Not  Mastered 


State  No. 

e 

Frequency 

Attributes  not  Mastered 

0* 

5.00 

19 

(all  mastered) 

1 

.34 

2 

12 

3 

.33 

8 

1.6 

10 

.40 

5 

4 

11 

.27 

2 

4.6 

12 

.00 

2 

1.  4.  6 

14 

-.12 

2 

1.  4.  6.  12 

59 

-.01 

5 

11 

61 

-.15 

2 

6.  11 

73 

-.66 

2 

1.  4.  6.  11.  12 

213 

-1.79 

2 

1,4.5,  6,  7,  8,9,  11,  12 

234 

.51 

2 

3 

237 

.10 

2 

1.3,6 

244 

.09 

5 

3,4 

246 

-.11 

4 

1.  3,  4,  6 

285 

-.48 

2 

1.  3,  6,  11 

293 

-.56 

2 

3,4,  6,  11 

294 

-.65 

2 

1.  3,  4,  6,  11 

304 

-1.01 

2 

1,  3,  4,  6,  8,  11 

336 

-.15 

2 

3,  5,  6,  9 

394 

-.57 

3 

3,  9,  11,  12 

453* 

-5.00 

6 

1,  2,  3,  4,  5,  6,  7,  8,  9,  10,  11,  12,  13  (none  mastered) 

*  Students  in  these  states  were  not  included  in  the  analysis  since  their  method  could  not  be 
determined. 
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Appendix  A 

The  Incidence  Matrix  for  Method  I  for  the  32  Items  Using  14  Attributes  with  Percent  Correa 
for  each  item  and  Percent  Mastered  for  each  Attribute. 


Item 

Attributes 

%  Correct 

11111 

12345678901234 

Method  I 

Total  sample 

1 

3+x=6+3*2 

01100000000000 

72 

74 

2 

7x+7=14 

01000001000000 

83 

81 

3 

16x=4 

00000000010000 

57 

63 

4 

6x=2x+3 

01000100010100 

58 

63 

5 

4x+21=10x+17 

0100010001 1101 

52 

60 

6 

35=7x 

00000000101000 

95 

93 

7 

8-f4(x-3)=24 

1 1010000100000 

67 

73 

8 

3-t-6x»18 

01000000100000 

69 

77 

9 

60+12=6x+2x 

00000100101000 

88 

81 

10 

4(2x->-3)=10x 

01010100101101 

84 

83 

11 

6-wtx-i-x:»22 

01000100100000 

75 

77 

12 

98=7+7x 

01000000101000 

85 

83 

13 

x-4=4+2*4 

10100000000000 

71 

73 

14 

llx-3x+4x=44-12+4 

00000100100000 

87 

87 

15 

4x+2=5+3x 

01000100000100 

84 

84 

16 

2+2*3(2x+3)=22x 

01110100101111 

27 

48 

17 

6x-4-8x=48-k48 

00000100100000 

79 

81 

18 

8-k4x=:26 

01000000100000 

83 

85 

19 

6(x+3)=12x 

01010100101101 

80 

81 

20 

5+3x+x=16 

01000100100000 

75 

76 

21 

3+2*2(2x-3)=23x 

11111110011111 

20 

42 

22 

75=5+5x 

01000000101000 

85 

84 

23 

24=6x 

00000000101000 

95 

92 

24 

12x+12=24 

01000001000000 

83 

81 

25 

4+x=6+2*3 

01 100000000000 

74 

73 

26 

8x»4X-)-2 

01000100010100 

66 

68 

27 

28x=7 

00000000010000 

53 

54 

28 

14x+30=78-2x 

1 1000100100000 

80 

78 

29 

5x+2x-3x=25+12-9 

00000100100000 

91 

88 

30 

x-6=3+5*3 

1 0 100000000000 

65 

67 

31 

7+4x=28x 

0100010001 1101 

47 

53 

32 

6-»4(x-2)=18 

1 1010000100000 

66 

70 

%MaSterBd  69665905959827 

44498519615937 
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Appendix  B 

The  Incidence  Matrix  for  Method  n  for  the  32  Items  Using  13  Attributes  with  Percent  Correct  for 
each  item  and  Percent  Mastered  for  each  Attribute. 


Item  Attributes  %  Correct 


nil 

1234567890123  Method n  Total  sample 


1 

3+x=6+3*2 

0110000000000 

73 

74 

2 

7x+7=14 

0100000010000 

80 

81 

3 

16x=4 

0000000000100 

63 

63 

4 

6x=2x+3 

0000001000100 

62 

63 

5 

4x+21=10x+17 

0100001100101 

61 

60 

6 

35=7x 

0100000001001 

97 

93 

7 

844(x-3)=24 

10011 lOlOlOOO 

76 

73 

8 

3-)'6x=18 

OlOOOOOOOlOOO 

85 

77 

9 

60+12=6x->-2x 

0100001101001 

81 

81 

10 

4(2x+3)=10x 

0101001101001 

84 

83 

11 

6-»4x+x»22 

0100001001000 

79 

77 

12 

98=7+7x 

0100000101001 

82 

83 

13 

x-4=4+2*4 

1010000000010 

73 

73 

14 

llx-3x-f4x»44-1244 

0000001001010 

90 

87 

15 

4x+2=5+3x 

0100001000000 

85 

84 

16 

2+2*3(2x+3)=22x 

0111001101011 

62 

48 

17 

6x+8x=48+48 

OOOOOOIOOIOOO 

82 

81 

18 

8-i4x=26 

OlOOOOOOOlOOO 

89 

85 

19 

6(x+3)=12x 

0101001 101001 

83 

81 

20 

5+3x+x=16 

0100001001000 

76 

76 

21 

3+2*2(2x-3)=23x 

iioniiioioii 

52 

42 

22 

75=5+5x 

0100000101001 

85 

84 

23 

24»6x 

0100000001001 

92 

92 

24 

I2x+12=24 

0100000010000 

79 

81 

25 

4+x=6+2*3 

01 10000000000 

70 

73 

26 

8x=4X+2 

0100001000100 

65 

68 

27 

28x=7 

0000000000100 

44 

54 

28 

14x+30=78-2x 

1  100001001000 

73 

78 

29 

5x+2x-3x=25+12-9 

0000001001000 

87 

88 

30 

x-6=3+5*3 

1010000000010 

66 

67 

31 

7+4x=28x 

0100001 100101 

52 

53 

32 

6-f4(x-2)=18 

0001 1 10001000 

71 

70 

1  1 
5064729870579 
1012681590728 


%  Mastered 
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Figure  Captions 

Figure  1 

A  Tree  Representation  of  the  States  in  Method  I  to  Which  More  Than  One  Student  Was  Classified 
Note;  The  small  numerals  correspond  to  the  State  labels. 

Figure  2 

A  Tree  Representatitm  of  the  States  in  Method  n  to  Which  More  Than  One  Student  Was  Classified 
Note;  The  small  numerals  correspond  to  the  State  labels. 
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